{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Add TFLiteNMS Plugin.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "machine_shape": "hm",
      "mount_file_id": "1IrhK5ruCugsff16qAAkYGg8PLoFFvmUW",
      "authorship_tag": "ABX9TyODnmNAOUt0VQ1WrZXzni0V",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/NobuoTsukamoto/tensorrt-examples/blob/main/python/detection/Add_TFLiteNMS_Plugin.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f598iir_p1U4"
      },
      "source": [
        "# Convert a SSDLite MobileNet V2 TFLite model to ONNX and Add TFLite NMS Plugin.\n",
        "\n",
        "This notebook converts from TensorFlow Lite model to ONNX model and replaces NonMax Suppression with TensorRT's TF-Lite NMS Plugin.  \n",
        "The model uses [SSD Lite MobileNet V2](https://github.com/tensorflow/models/blob/a4fd64722dcdd42361beb1be478ad8fdb10bde31/research/object_detection/g3doc/tf1_detection_zoo.md) and modifies the ONNX model with [ONNX GraphSurgeon](https://github.com/NVIDIA/TensorRT/tree/master/tools/onnx-graphsurgeon)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "heEyNGCd-dgM"
      },
      "source": [
        "Copyright (c) 2021 Nobuo Tsukamoto  \n",
        "  \n",
        "This software is released under the MIT License.  \n",
        "See the LICENSE file in the project root for more information.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6VghN88vg2NM"
      },
      "source": [
        "## Reference\n",
        "- [onnx/tensorflow-onnx - Convert a mobiledet tflite model to ONNX](https://github.com/onnx/tensorflow-onnx/blob/de67f2051d1f036b29901e2910c8eb41a1c71b6e/tutorials/mobiledet-tflite.ipynb)\n",
        "- [TensorFlow Model Garden - TensorFlow 1 Detection Model Zoo](https://github.com/tensorflow/models/blob/a4fd64722dcdd42361beb1be478ad8fdb10bde31/research/object_detection/g3doc/tf1_detection_zoo.md)\n",
        "- [TensorFlow Model Garden - Running on mobile with TensorFlow Lite](https://github.com/tensorflow/models/blob/a4fd64722dcdd42361beb1be478ad8fdb10bde31/research/object_detection/g3doc/running_on_mobile_tensorflowlite.md)\n",
        "- [TensorRT Backend For ONNX](https://github.com/onnx/onnx-tensorrt/tree/868e636f51f0d7e61df340371303275265146fe0)\n",
        "- [ONNX GraphSurgeon](https://github.com/NVIDIA/TensorRT/tree/0953f2ff8762b28e0f1bef0582b6ca3d7a12fcaa/tools/onnx-graphsurgeon)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "92EB-Un1htTK"
      },
      "source": [
        "## Convert the TensorFlow Lite Model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XDwBImKugaIU",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6e96a5b3-32dc-43e8-e388-c262f10105a8"
      },
      "source": [
        "%tensorflow_version 1.x"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "TensorFlow 1.x selected.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EEd6ynRMiQLJ"
      },
      "source": [
        "Clone [tensorflow/models](https://github.com/tensorflow/models) repository and install dependency."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8qHlK2mdgp4X",
        "outputId": "fcec66f7-fe20-4e84-97c5-aa0c8c7720be"
      },
      "source": [
        "%%bash\n",
        "\n",
        "pip install tensorflow-addons\n",
        "git clone https://github.com/tensorflow/models.git\n",
        "cd models\n",
        "git checkout a4fd64722dcdd42361beb1be478ad8fdb10bde31\n",
        "\n",
        "cd research\n",
        "protoc object_detection/protos/*.proto --python_out=.\n",
        "\n",
        "cp object_detection/packages/tf1/setup.py .\n",
        "python -m pip install  ."
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting tensorflow-addons\n",
            "  Downloading https://files.pythonhosted.org/packages/66/4b/e893d194e626c24b3df2253066aa418f46a432fdb68250cde14bf9bb0700/tensorflow_addons-0.13.0-cp37-cp37m-manylinux2010_x86_64.whl (679kB)\n",
            "Requirement already satisfied: typeguard>=2.7 in /usr/local/lib/python3.7/dist-packages (from tensorflow-addons) (2.7.1)\n",
            "Installing collected packages: tensorflow-addons\n",
            "Successfully installed tensorflow-addons-0.13.0\n",
            "Processing /content/models/research\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (7.1.2)\n",
            "Requirement already satisfied: lxml in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (4.2.6)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (3.2.2)\n",
            "Requirement already satisfied: Cython in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (0.29.23)\n",
            "Requirement already satisfied: contextlib2 in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (0.5.5)\n",
            "Collecting tf-slim\n",
            "  Downloading https://files.pythonhosted.org/packages/02/97/b0f4a64df018ca018cc035d44f2ef08f91e2e8aa67271f6f19633a015ff7/tf_slim-1.1.0-py2.py3-none-any.whl (352kB)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (1.15.0)\n",
            "Requirement already satisfied: pycocotools in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (2.0.2)\n",
            "Collecting lvis\n",
            "  Downloading https://files.pythonhosted.org/packages/72/b6/1992240ab48310b5360bfdd1d53163f43bb97d90dc5dc723c67d41c38e78/lvis-0.5.3-py3-none-any.whl\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (1.4.1)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (1.1.5)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->object-detection==0.1) (2.4.7)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->object-detection==0.1) (2.8.1)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->object-detection==0.1) (1.3.1)\n",
            "Requirement already satisfied: numpy>=1.11 in /usr/local/lib/python3.7/dist-packages (from matplotlib->object-detection==0.1) (1.19.5)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->object-detection==0.1) (0.10.0)\n",
            "Requirement already satisfied: absl-py>=0.2.2 in /usr/local/lib/python3.7/dist-packages (from tf-slim->object-detection==0.1) (0.12.0)\n",
            "Requirement already satisfied: setuptools>=18.0 in /usr/local/lib/python3.7/dist-packages (from pycocotools->object-detection==0.1) (57.0.0)\n",
            "Requirement already satisfied: opencv-python>=4.1.0.25 in /usr/local/lib/python3.7/dist-packages (from lvis->object-detection==0.1) (4.1.2.30)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->object-detection==0.1) (2018.9)\n",
            "Building wheels for collected packages: object-detection\n",
            "  Building wheel for object-detection (setup.py): started\n",
            "  Building wheel for object-detection (setup.py): finished with status 'done'\n",
            "  Created wheel for object-detection: filename=object_detection-0.1-cp37-none-any.whl size=1657627 sha256=7b19e961b8a0662585209cca8398fe4e5a8519ba2561ae264fb607497b10fa2b\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-omepu6j0/wheels/94/49/4b/39b051683087a22ef7e80ec52152a27249d1a644ccf4e442ea\n",
            "Successfully built object-detection\n",
            "Installing collected packages: tf-slim, lvis, object-detection\n",
            "Successfully installed lvis-0.5.3 object-detection-0.1 tf-slim-1.1.0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'models'...\n",
            "Note: checking out 'a4fd64722dcdd42361beb1be478ad8fdb10bde31'.\n",
            "\n",
            "You are in 'detached HEAD' state. You can look around, make experimental\n",
            "changes and commit them, and you can discard any commits you make in this\n",
            "state without impacting any branches by performing another checkout.\n",
            "\n",
            "If you want to create a new branch to retain commits you create, you may\n",
            "do so (now or later) by using -b with the checkout command again. Example:\n",
            "\n",
            "  git checkout -b <new-branch-name>\n",
            "\n",
            "HEAD is now at a4fd6472 Internal change\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qJiVt0nbgxYj",
        "outputId": "5a61c165-504f-4a6e-ed89-ec3a1b09fa94"
      },
      "source": [
        "import os\n",
        "\n",
        "os.environ['PYTHONPATH'] = '/content/models:' + os.environ['PYTHONPATH']\n",
        "print(os.environ['PYTHONPATH'])"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/models:/tensorflow-1.15.2/python3.7:/env/python\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "I5PPsFUUhNAQ",
        "outputId": "a01e0bfe-d3c8-42cc-8d7d-bde9cbc7a2ef"
      },
      "source": [
        "%cd models/research/\n",
        "!python object_detection/builders/model_builder_tf1_test.py"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/models/research\n",
            "Running tests under Python 3.7.10: /usr/bin/python3\n",
            "[ RUN      ] ModelBuilderTF1Test.test_create_context_rcnn_from_config_with_params0 (True)\n",
            "[       OK ] ModelBuilderTF1Test.test_create_context_rcnn_from_config_with_params0 (True)\n",
            "[ RUN      ] ModelBuilderTF1Test.test_create_context_rcnn_from_config_with_params1 (False)\n",
            "[       OK ] ModelBuilderTF1Test.test_create_context_rcnn_from_config_with_params1 (False)\n",
            "[ RUN      ] ModelBuilderTF1Test.test_create_experimental_model\n",
            "[       OK ] ModelBuilderTF1Test.test_create_experimental_model\n",
            "[ RUN      ] ModelBuilderTF1Test.test_create_faster_rcnn_from_config_with_crop_feature0 (True)\n",
            "[       OK ] ModelBuilderTF1Test.test_create_faster_rcnn_from_config_with_crop_feature0 (True)\n",
            "[ RUN      ] ModelBuilderTF1Test.test_create_faster_rcnn_from_config_with_crop_feature1 (False)\n",
            "[       OK ] ModelBuilderTF1Test.test_create_faster_rcnn_from_config_with_crop_feature1 (False)\n",
            "[ RUN      ] ModelBuilderTF1Test.test_create_faster_rcnn_model_from_config_with_example_miner\n",
            "[       OK ] ModelBuilderTF1Test.test_create_faster_rcnn_model_from_config_with_example_miner\n",
            "[ RUN      ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_faster_rcnn_with_matmul\n",
            "[       OK ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_faster_rcnn_with_matmul\n",
            "[ RUN      ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_faster_rcnn_without_matmul\n",
            "[       OK ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_faster_rcnn_without_matmul\n",
            "[ RUN      ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_mask_rcnn_with_matmul\n",
            "[       OK ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_mask_rcnn_with_matmul\n",
            "[ RUN      ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_mask_rcnn_without_matmul\n",
            "[       OK ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_mask_rcnn_without_matmul\n",
            "[ RUN      ] ModelBuilderTF1Test.test_create_rfcn_model_from_config\n",
            "[       OK ] ModelBuilderTF1Test.test_create_rfcn_model_from_config\n",
            "[ RUN      ] ModelBuilderTF1Test.test_create_ssd_fpn_model_from_config\n",
            "[       OK ] ModelBuilderTF1Test.test_create_ssd_fpn_model_from_config\n",
            "[ RUN      ] ModelBuilderTF1Test.test_create_ssd_models_from_config\n",
            "[       OK ] ModelBuilderTF1Test.test_create_ssd_models_from_config\n",
            "[ RUN      ] ModelBuilderTF1Test.test_invalid_faster_rcnn_batchnorm_update\n",
            "[       OK ] ModelBuilderTF1Test.test_invalid_faster_rcnn_batchnorm_update\n",
            "[ RUN      ] ModelBuilderTF1Test.test_invalid_first_stage_nms_iou_threshold\n",
            "[       OK ] ModelBuilderTF1Test.test_invalid_first_stage_nms_iou_threshold\n",
            "[ RUN      ] ModelBuilderTF1Test.test_invalid_model_config_proto\n",
            "[       OK ] ModelBuilderTF1Test.test_invalid_model_config_proto\n",
            "[ RUN      ] ModelBuilderTF1Test.test_invalid_second_stage_batch_size\n",
            "[       OK ] ModelBuilderTF1Test.test_invalid_second_stage_batch_size\n",
            "[ RUN      ] ModelBuilderTF1Test.test_session\n",
            "[  SKIPPED ] ModelBuilderTF1Test.test_session\n",
            "[ RUN      ] ModelBuilderTF1Test.test_unknown_faster_rcnn_feature_extractor\n",
            "[       OK ] ModelBuilderTF1Test.test_unknown_faster_rcnn_feature_extractor\n",
            "[ RUN      ] ModelBuilderTF1Test.test_unknown_meta_architecture\n",
            "[       OK ] ModelBuilderTF1Test.test_unknown_meta_architecture\n",
            "[ RUN      ] ModelBuilderTF1Test.test_unknown_ssd_feature_extractor\n",
            "[       OK ] ModelBuilderTF1Test.test_unknown_ssd_feature_extractor\n",
            "----------------------------------------------------------------------\n",
            "Ran 21 tests in 0.224s\n",
            "\n",
            "OK (skipped=1)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kvLFSF2hivaA"
      },
      "source": [
        "Download SSDLite MobileNet V2 checkpoint and export TensorFlow Lite model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AZUJyMjEm5d6",
        "outputId": "028b8817-a091-4bf2-dd64-024441e0a229"
      },
      "source": [
        "!wget http://download.tensorflow.org/models/object_detection/ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz -P /content\n",
        "!tar xf /content/ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz -C /content"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-06-30 11:09:23--  http://download.tensorflow.org/models/object_detection/ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz\n",
            "Resolving download.tensorflow.org (download.tensorflow.org)... 108.177.126.128, 2a00:1450:4013:c01::80\n",
            "Connecting to download.tensorflow.org (download.tensorflow.org)|108.177.126.128|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 51025348 (49M) [application/x-tar]\n",
            "Saving to: ‘/content/ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz.1’\n",
            "\n",
            "ssdlite_mobilenet_v 100%[===================>]  48.66M   155MB/s    in 0.3s    \n",
            "\n",
            "2021-06-30 11:09:23 (155 MB/s) - ‘/content/ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz.1’ saved [51025348/51025348]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fJEzEHIMgWhV",
        "outputId": "950dd63f-9e5b-4e5f-846d-79434f8edcc3"
      },
      "source": [
        "!python object_detection/export_tflite_ssd_graph.py \\\n",
        "    --pipeline_config_path=\"/content/ssdlite_mobilenet_v2_coco_2018_05_09/pipeline.config\" \\\n",
        "    --trained_checkpoint_prefix=\"/content/ssdlite_mobilenet_v2_coco_2018_05_09/model.ckpt\" \\\n",
        "    --output_directory=\"/content/ssdlite_mobilenet_v2_coco_2018_05_09/tflite\" \\\n",
        "    --add_postprocessing_op=true"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tf_slim/layers/layers.py:1089: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use `layer.__call__` method instead.\n",
            "W0630 11:09:33.057559 139931898357632 deprecation.py:323] From /usr/local/lib/python3.7/dist-packages/tf_slim/layers/layers.py:1089: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use `layer.__call__` method instead.\n",
            "INFO:tensorflow:depth of additional conv before box predictor: 0\n",
            "I0630 11:09:35.229738 139931898357632 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
            "INFO:tensorflow:depth of additional conv before box predictor: 0\n",
            "I0630 11:09:35.306802 139931898357632 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
            "INFO:tensorflow:depth of additional conv before box predictor: 0\n",
            "I0630 11:09:35.391663 139931898357632 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
            "INFO:tensorflow:depth of additional conv before box predictor: 0\n",
            "I0630 11:09:35.473502 139931898357632 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
            "INFO:tensorflow:depth of additional conv before box predictor: 0\n",
            "I0630 11:09:35.551683 139931898357632 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
            "INFO:tensorflow:depth of additional conv before box predictor: 0\n",
            "I0630 11:09:35.628433 139931898357632 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
            "2021-06-30 11:09:35.715751: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1\n",
            "2021-06-30 11:09:35.793341: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-06-30 11:09:35.793934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
            "name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53\n",
            "pciBusID: 0000:00:04.0\n",
            "2021-06-30 11:09:35.812085: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
            "2021-06-30 11:09:36.048447: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
            "2021-06-30 11:09:36.160969: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
            "2021-06-30 11:09:36.186427: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
            "2021-06-30 11:09:36.437209: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
            "2021-06-30 11:09:36.468568: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
            "2021-06-30 11:09:36.943688: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
            "2021-06-30 11:09:36.943873: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-06-30 11:09:36.944590: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-06-30 11:09:36.945123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
            "2021-06-30 11:09:36.945553: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX512F\n",
            "2021-06-30 11:09:36.964441: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2000160000 Hz\n",
            "2021-06-30 11:09:36.964802: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x56040f9c6d80 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n",
            "2021-06-30 11:09:36.964835: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version\n",
            "2021-06-30 11:09:37.086796: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-06-30 11:09:37.087659: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x56040f9c7640 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:\n",
            "2021-06-30 11:09:37.087697: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla V100-SXM2-16GB, Compute Capability 7.0\n",
            "2021-06-30 11:09:37.087865: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-06-30 11:09:37.088445: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
            "name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53\n",
            "pciBusID: 0000:00:04.0\n",
            "2021-06-30 11:09:37.088510: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
            "2021-06-30 11:09:37.088529: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
            "2021-06-30 11:09:37.088543: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
            "2021-06-30 11:09:37.088557: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
            "2021-06-30 11:09:37.088570: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
            "2021-06-30 11:09:37.088583: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
            "2021-06-30 11:09:37.088597: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
            "2021-06-30 11:09:37.088659: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-06-30 11:09:37.089217: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-06-30 11:09:37.089728: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
            "2021-06-30 11:09:37.093136: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
            "2021-06-30 11:09:37.094557: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
            "2021-06-30 11:09:37.094587: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 \n",
            "2021-06-30 11:09:37.094595: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N \n",
            "2021-06-30 11:09:37.095571: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-06-30 11:09:37.096302: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-06-30 11:09:37.096925: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
            "2021-06-30 11:09:37.096963: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15060 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:04.0, compute capability: 7.0)\n",
            "WARNING:tensorflow:From /tensorflow-1.15.2/python3.7/tensorflow_core/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file APIs to check for files with this prefix.\n",
            "W0630 11:09:37.595122 139931898357632 deprecation.py:323] From /tensorflow-1.15.2/python3.7/tensorflow_core/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file APIs to check for files with this prefix.\n",
            "2021-06-30 11:09:38.100181: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-06-30 11:09:38.100783: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
            "name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53\n",
            "pciBusID: 0000:00:04.0\n",
            "2021-06-30 11:09:38.100851: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
            "2021-06-30 11:09:38.100869: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
            "2021-06-30 11:09:38.100882: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
            "2021-06-30 11:09:38.100897: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
            "2021-06-30 11:09:38.100910: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
            "2021-06-30 11:09:38.100923: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
            "2021-06-30 11:09:38.100937: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
            "2021-06-30 11:09:38.100999: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-06-30 11:09:38.101599: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-06-30 11:09:38.102107: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
            "2021-06-30 11:09:38.102140: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
            "2021-06-30 11:09:38.102168: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 \n",
            "2021-06-30 11:09:38.102178: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N \n",
            "2021-06-30 11:09:38.102277: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-06-30 11:09:38.102841: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-06-30 11:09:38.103404: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15060 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:04.0, compute capability: 7.0)\n",
            "INFO:tensorflow:Restoring parameters from /content/ssdlite_mobilenet_v2_coco_2018_05_09/model.ckpt\n",
            "I0630 11:09:38.104365 139931898357632 saver.py:1284] Restoring parameters from /content/ssdlite_mobilenet_v2_coco_2018_05_09/model.ckpt\n",
            "WARNING:tensorflow:From /tensorflow-1.15.2/python3.7/tensorflow_core/python/tools/freeze_graph.py:233: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.compat.v1.graph_util.convert_variables_to_constants`\n",
            "W0630 11:09:42.349626 139931898357632 deprecation.py:323] From /tensorflow-1.15.2/python3.7/tensorflow_core/python/tools/freeze_graph.py:233: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.compat.v1.graph_util.convert_variables_to_constants`\n",
            "WARNING:tensorflow:From /tensorflow-1.15.2/python3.7/tensorflow_core/python/framework/graph_util_impl.py:277: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.compat.v1.graph_util.extract_sub_graph`\n",
            "W0630 11:09:42.349879 139931898357632 deprecation.py:323] From /tensorflow-1.15.2/python3.7/tensorflow_core/python/framework/graph_util_impl.py:277: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.compat.v1.graph_util.extract_sub_graph`\n",
            "INFO:tensorflow:Froze 404 variables.\n",
            "I0630 11:09:42.673049 139931898357632 graph_util_impl.py:334] Froze 404 variables.\n",
            "INFO:tensorflow:Converted 404 variables to const ops.\n",
            "I0630 11:09:42.726873 139931898357632 graph_util_impl.py:394] Converted 404 variables to const ops.\n",
            "2021-06-30 11:09:42.818043: I tensorflow/tools/graph_transforms/transform_graph.cc:317] Applying strip_unused_nodes\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5z0bcYh4iE8a",
        "outputId": "19c31a5d-64d6-4176-dbf7-be0f7d7968b2"
      },
      "source": [
        "!tflite_convert \\\n",
        "    --enable_v1_converter \\\n",
        "    --graph_def_file=\"/content/ssdlite_mobilenet_v2_coco_2018_05_09/tflite/tflite_graph.pb\" \\\n",
        "    --output_file=\"/content/ssdlite_mobilenet_v2_coco_2018_05_09/tflite/ssdlite_mobilenet_v2_320x320.tflite\" \\\n",
        "    --inference_input_type=FLOAT \\\n",
        "    --inference_type=FLOAT \\\n",
        "    --input_arrays=\"normalized_input_image_tensor\" \\\n",
        "    --output_arrays=\"TFLite_Detection_PostProcess,TFLite_Detection_PostProcess:1,TFLite_Detection_PostProcess:2,TFLite_Detection_PostProcess:3\" \\\n",
        "    --input_shapes=1,300,300,3 \\\n",
        "    --allow_nudging_weights_to_use_fast_gemm_kernel=true \\\n",
        "    --allow_custom_op"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2021-06-30 11:09:53.163088: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1\n",
            "2021-06-30 11:09:53.184290: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-06-30 11:09:53.184868: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
            "name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53\n",
            "pciBusID: 0000:00:04.0\n",
            "2021-06-30 11:09:53.185132: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
            "2021-06-30 11:09:53.186718: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
            "2021-06-30 11:09:53.194978: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
            "2021-06-30 11:09:53.195323: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
            "2021-06-30 11:09:53.197130: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
            "2021-06-30 11:09:53.198057: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
            "2021-06-30 11:09:53.201640: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
            "2021-06-30 11:09:53.201761: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-06-30 11:09:53.202392: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-06-30 11:09:53.202971: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
            "2021-06-30 11:09:53.203325: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX512F\n",
            "2021-06-30 11:09:53.207982: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2000160000 Hz\n",
            "2021-06-30 11:09:53.208285: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x559f83812d80 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n",
            "2021-06-30 11:09:53.208314: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version\n",
            "2021-06-30 11:09:53.297878: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-06-30 11:09:53.298721: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x559f83813b80 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:\n",
            "2021-06-30 11:09:53.298760: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla V100-SXM2-16GB, Compute Capability 7.0\n",
            "2021-06-30 11:09:53.298934: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-06-30 11:09:53.299559: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
            "name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53\n",
            "pciBusID: 0000:00:04.0\n",
            "2021-06-30 11:09:53.299632: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
            "2021-06-30 11:09:53.299657: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
            "2021-06-30 11:09:53.299677: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
            "2021-06-30 11:09:53.299703: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
            "2021-06-30 11:09:53.299720: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
            "2021-06-30 11:09:53.299741: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
            "2021-06-30 11:09:53.299763: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
            "2021-06-30 11:09:53.299840: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-06-30 11:09:53.300478: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-06-30 11:09:53.301029: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
            "2021-06-30 11:09:53.301095: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
            "2021-06-30 11:09:53.302370: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
            "2021-06-30 11:09:53.302408: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 \n",
            "2021-06-30 11:09:53.302418: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N \n",
            "2021-06-30 11:09:53.302521: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-06-30 11:09:53.303131: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-06-30 11:09:53.303705: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
            "2021-06-30 11:09:53.303746: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15060 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:04.0, compute capability: 7.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DfAP2oH6qTaU"
      },
      "source": [
        "## Export ONNX Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "faSTJYvxnKch"
      },
      "source": [
        "Install dependency."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "knvpxQlzjfOP",
        "outputId": "b72de806-2193-479d-8150-4eb9395688c6"
      },
      "source": [
        "%%bash\n",
        "pip3 install onnxruntime\n",
        "pip3 install tf2onnx"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting onnxruntime\n",
            "  Downloading https://files.pythonhosted.org/packages/f9/76/3d0f8bb2776961c7335693df06eccf8d099e48fa6fb552c7546867192603/onnxruntime-1.8.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.5MB)\n",
            "Requirement already satisfied: flatbuffers in /usr/local/lib/python3.7/dist-packages (from onnxruntime) (1.12)\n",
            "Requirement already satisfied: protobuf in /usr/local/lib/python3.7/dist-packages (from onnxruntime) (3.12.4)\n",
            "Requirement already satisfied: numpy>=1.16.6 in /usr/local/lib/python3.7/dist-packages (from onnxruntime) (1.19.5)\n",
            "Requirement already satisfied: six>=1.9 in /usr/local/lib/python3.7/dist-packages (from protobuf->onnxruntime) (1.15.0)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from protobuf->onnxruntime) (57.0.0)\n",
            "Installing collected packages: onnxruntime\n",
            "Successfully installed onnxruntime-1.8.0\n",
            "Collecting tf2onnx\n",
            "  Downloading https://files.pythonhosted.org/packages/9e/dc/a87fe3ff16092a0bca63ce24bcc086c0f2abe7decb057829669f99a7565e/tf2onnx-1.8.5-py3-none-any.whl (370kB)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from tf2onnx) (1.15.0)\n",
            "Collecting onnx>=1.4.1\n",
            "  Downloading https://files.pythonhosted.org/packages/3f/9b/54c950d3256e27f970a83cd0504efb183a24312702deed0179453316dbd0/onnx-1.9.0-cp37-cp37m-manylinux2010_x86_64.whl (12.2MB)\n",
            "Requirement already satisfied: numpy>=1.14.1 in /usr/local/lib/python3.7/dist-packages (from tf2onnx) (1.19.5)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from tf2onnx) (2.23.0)\n",
            "Requirement already satisfied: flatbuffers~=1.12 in /usr/local/lib/python3.7/dist-packages (from tf2onnx) (1.12)\n",
            "Requirement already satisfied: typing-extensions>=3.6.2.1 in /usr/local/lib/python3.7/dist-packages (from onnx>=1.4.1->tf2onnx) (3.7.4.3)\n",
            "Requirement already satisfied: protobuf in /usr/local/lib/python3.7/dist-packages (from onnx>=1.4.1->tf2onnx) (3.12.4)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->tf2onnx) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->tf2onnx) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->tf2onnx) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->tf2onnx) (2021.5.30)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from protobuf->onnx>=1.4.1->tf2onnx) (57.0.0)\n",
            "Installing collected packages: onnx, tf2onnx\n",
            "Successfully installed onnx-1.9.0 tf2onnx-1.8.5\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I10zwTrynWYJ"
      },
      "source": [
        "Note: TensorRT 7.2 supports operators up to Opset 13.  \n",
        "- [onnx/onnx-tensorrt - Supported ONNX Operators](https://github.com/onnx/onnx-tensorrt/blob/868e636f51f0d7e61df340371303275265146fe0/docs/operators.md)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fbTs3y5nj_dX",
        "outputId": "e2bc164e-4802-4918-b2a0-7ff83f15c1fd"
      },
      "source": [
        "!python3 -m tf2onnx.convert --opset 11 \\\n",
        "    --tflite /content/ssdlite_mobilenet_v2_coco_2018_05_09/tflite/ssdlite_mobilenet_v2_320x320.tflite \\\n",
        "    --output /content/ssdlite_mobilenet_v2_coco_2018_05_09/onnx/ssdlite_mobilenet_v2_320x320.onnx"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/lib/python3.7/runpy.py:125: RuntimeWarning: 'tf2onnx.convert' found in sys.modules after import of package 'tf2onnx', but prior to execution of 'tf2onnx.convert'; this may result in unpredictable behaviour\n",
            "  warn(RuntimeWarning(msg))\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tf2onnx/verbose_logging.py:76: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.\n",
            "\n",
            "2021-06-30 11:18:46,649 - WARNING - From /usr/local/lib/python3.7/dist-packages/tf2onnx/verbose_logging.py:76: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.\n",
            "\n",
            "2021-06-30 11:18:46,650 - INFO - Using tensorflow=1.15.2, onnx=1.9.0, tf2onnx=1.8.5/50049d\n",
            "2021-06-30 11:18:46,650 - INFO - Using opset <onnx, 13>\n",
            "2021-06-30 11:18:47,078 - WARNING - NMS node TFLite_Detection_PostProcess uses fast NMS. ONNX will approximate with standard NMS.\n",
            "2021-06-30 11:18:47,112 - INFO - Optimizing ONNX model\n",
            "2021-06-30 11:18:48,579 - INFO - After optimization: Cast -13 (15->2), Const -180 (377->197), Identity -4 (4->0), Reshape -33 (46->13), Transpose -341 (353->12)\n",
            "2021-06-30 11:18:48,622 - INFO - \n",
            "2021-06-30 11:18:48,622 - INFO - Successfully converted TensorFlow model /content/ssdlite_mobilenet_v2_coco_2018_05_09/tflite/ssdlite_mobilenet_v2_320x320.tflite to ONNX\n",
            "2021-06-30 11:18:48,622 - INFO - Model inputs: ['normalized_input_image_tensor']\n",
            "2021-06-30 11:18:48,622 - INFO - Model outputs: ['TFLite_Detection_PostProcess', 'TFLite_Detection_PostProcess:1', 'TFLite_Detection_PostProcess:2', 'TFLite_Detection_PostProcess:3']\n",
            "2021-06-30 11:18:48,622 - INFO - ONNX model is saved at /content/ssdlite_mobilenet_v2_coco_2018_05_09/onnx/ssdlite_mobilenet_v2_320x320.onnx\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KN5E-Q5uqo3y"
      },
      "source": [
        "## Add TF-Lite NMS Plugin"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kCQ1qTMvk1nc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2d386cdb-d2ff-4769-cd67-cdaf8a014901"
      },
      "source": [
        "!python3 -m pip install onnx_graphsurgeon --index-url https://pypi.ngc.nvidia.com"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Looking in indexes: https://pypi.ngc.nvidia.com\n",
            "Collecting onnx_graphsurgeon\n",
            "  Downloading https://developer.download.nvidia.com/compute/redist/onnx-graphsurgeon/onnx_graphsurgeon-0.3.9-py2.py3-none-any.whl\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from onnx_graphsurgeon) (1.19.5)\n",
            "Requirement already satisfied: onnx in /usr/local/lib/python3.7/dist-packages (from onnx_graphsurgeon) (1.9.0)\n",
            "Requirement already satisfied: protobuf in /usr/local/lib/python3.7/dist-packages (from onnx->onnx_graphsurgeon) (3.12.4)\n",
            "Requirement already satisfied: typing-extensions>=3.6.2.1 in /usr/local/lib/python3.7/dist-packages (from onnx->onnx_graphsurgeon) (3.7.4.3)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from onnx->onnx_graphsurgeon) (1.15.0)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from protobuf->onnx->onnx_graphsurgeon) (57.0.0)\n",
            "Installing collected packages: onnx-graphsurgeon\n",
            "Successfully installed onnx-graphsurgeon-0.3.9\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "USJqsm-7lRK1",
        "outputId": "6d8b59e3-adfd-4149-c739-0a87a7f75e45"
      },
      "source": [
        "%cd /content/\n",
        "!git clone https://github.com/NobuoTsukamoto/tensorrt-examples"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content\n",
            "Cloning into 'tensorrt-examples'...\n",
            "remote: Enumerating objects: 17, done.\u001b[K\n",
            "remote: Counting objects: 100% (17/17), done.\u001b[K\n",
            "remote: Compressing objects: 100% (12/12), done.\u001b[K\n",
            "remote: Total 17 (delta 1), reused 13 (delta 1), pack-reused 0\u001b[K\n",
            "Unpacking objects: 100% (17/17), done.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EZSIts0rlciG",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "58d0957f-907d-439c-b356-b8371c4a76d5"
      },
      "source": [
        "%cd /content/tensorrt-examples/python/detection"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/tensorrt-examples/python/detection\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2nOXo6yJnfD-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7cd6c371-d8b2-40ae-c56d-234ffcf0921b"
      },
      "source": [
        "!python3 add_tensorrt_tflitenms_plugin.py \\\n",
        "    --input /content/ssdlite_mobilenet_v2_coco_2018_05_09/onnx/ssdlite_mobilenet_v2_320x320.onnx \\\n",
        "    --output /content/ssdlite_mobilenet_v2_coco_2018_05_09/onnx/ssdlite_mobilenet_v2_320x320_gs.onnx"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{'max_classes_per_detection': 1, 'max_detections': 10, 'back_ground_Label_id': 0, 'nms_iou_threshold': 0.6, 'nms_score_threshold': 0.1, 'num_classes': 91, 'y_scale': 10.0, 'x_scale': 10.0, 'h_scale': 5.0, 'w_scale': 5.0, 'scoreBits': 16}\n",
            "Saving the ONNX model to /content/ssdlite_mobilenet_v2_coco_2018_05_09/onnx/ssdlite_mobilenet_v2_320x320_gs.onnx\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}